1. The Field of the Invention.
This invention relates generally to combining real scene elements from a television-type camera with virtual scene elements from a virtual camera. More specifically, the present invention provides new methods for generating a garbage matte which simplifies proper commingling of real and virtual scene elements so that they appear to be positioned and can be moved relative to each other by utilizing a universal blue/green screen compositor to generate a single picture.
2. The State of the Art
The state of the art of combining real world imagery with additional imagery provided by another source is a process requiring substantial precision. Typically, the other source of additional imagery is computer generated or some other type of synthetic imagery. Precision is required in the placement of synthetic imagery so that the combination appears realistic when viewed as a whole. The combination is typically referred to as a virtual set.
The virtual set to be discussed hereinafter comprises input from a television camera and a virtual camera. The virtual camera is typically implemented utilizing a high performance computer graphics engine, although speed can be sacrificed in achieving high quality output.
The type of computer generated imagery of concern to the present invention is imagery which can be inserted within a scene of real world imagery photographed or filmed by a television camera (where the television camera is generally a source of video data). In this scenario, real world objects are disposed in the foreground in front of a studio backdrop background, commonly constructed using a blue or green wall or screen. A synthetic scene is then added in place of the blue or green screen.
The virtual set described above involves important set-up, calibration and implementation details which are mostly beyond the scope of the present invention. However, to facilitate understanding, the subject matter of two previously filed patent applications titled "METHOD FOR MEASURING CAMERA AND LENS PROPERTIES FOR CAMERA TRACKING", and "CAMERA/LENS CALIBRATION APPARATUS AND METHOD", both of which were filed on Apr. 4, 1997, are hereinafter incorporated by reference.
The patent applications described above teach that as long as a position, an orientation, and a field of view of the physical television camera are monitored and duplicated by the virtual camera, the real and virtual scene elements appear to move together, and are properly "co-mingled" or combined in a virtual set. An issue that is not addressed, however, is the matter of masking objects. It is important to understand that those objects or other scene elements which the television camera can view but which are not to be part of a composited picture are covered from view of the television camera by utilizing a garbage matte.
Garbage mattes have been used for years to hide or mask objects from the field of view of the television camera, as will be explained. In the context of the present invention, physical scene elements are typically placed on a physical set which is painted blue or green. In the physical set, the floor, backdrop, and possibly side walls are painted a uniform blue. This is referred to as a blue stage. Physical scene elements typically include live talent (actors), chairs, desks, plants or any other "real" props with which the live talent can interact.
Virtual scene elements are built using computer modeling tools known to those skilled in the art. The computer modeling tools are utilized to "create" geometric shapes, texture patterns, lighting effects, and other special effects, in order to create a desired virtual environment. These virtual scene elements are stored as digital data until they are retrieved to become part of the virtual set. It should be mentioned that the position and orientation of each item in the virtual environment is modeled relative to a coordinate system selected for modeling.
It is an important realization that motivation for the present invention comes from a desire and need to often adjust the position and orientation of the physical blue stage relative to the virtual environment. In essence, the effect is to "grab" the physical set (including the television cameras) and "place" it in any desired position and orientation relative to the virtual environment.
In the process of combining the virtual environment (and thus the virtual set) with the physical set, a "tool" which is often employed by those skilled in the art is the previously mentioned garbage matte. The garbage matte receives its name from the function it performs of hiding undesirable physical scene elements which might at times be in the television camera's field of view, but which are not supposed to be seen within the composited output image (showing the virtual set instead).
It should be obvious that lights above a physical stage should not be included in the composited output image. However, because the stage lights are not blue, they will undesirably appear as part of the virtual set. Therefore, a garbage matte is utilized to eliminate the stage lights or other undesirable physical structures from the virtual set. If the garbage matte is "real," for example, it can be a piece of cardboard which is cut to the desired shape and painted blue.
However, in the case where the garbage matte is virtual, then it is often constructed in the virtual environment using one or more polygons generated by the computer modeling tools or some other polygon generating tool running in a computer image generator. The polygons indicate directions where the television camera's picture should not contribute to a composited picture. In other words, the polygons hide or mask physical objects. Therefore, in any direction where the garbage matte is seen in a compositor, the composited output image is taken from the virtual environment and not the physical set, regardless of whether the television camera sees blue or not.
A common example of the utilization of a garbage matte is the obvious case where the television camera pans beyond the limits of the blue stage, thus revealing lights, props, studio personnel, or other objects just beyond the walls of the physical set. Because the compositor "sees" these objects as not being blue, they would normally be passed to the composited output image. Therefore, without a garbage matte, it is not possible to pan the television camera beyond the limits of the physical set which has not been colored blue, or hidden behind blue walls. It should be apparent that even if a full wrap-around physical blue set were created, there would necessarily be overhead lights which should be masked off, requiring the use of a garbage matte.
The information above has been directed to use of the garbage matte. What has not been discussed is the creation of a garbage matte. The state of the art methods for creating and positioning an effective garbage matte are generally tedious and time consuming.
For example, it is common to construct a simple garbage matte polygon using a real two-dimensional polygon cutout. The shape of the cutout approximates the shape of the physical set blue stage as seen from a particular television camera location. Unfortunately, a two-dimensional cutout only works well from a narrow range of selectable television camera positions. Furthermore, the two-dimensional cutout can't obscure areas behind the television camera's position at all. Consequently, it is generally necessary to move the garbage matte to different positions and orientations for different television camera positions. It becomes readily apparent that the creation, moving and repositioning of garbage mattes can rapidly become a time consuming and thus costly procedure. Furthermore, the problem is only compounded if the television camera must be moved so as to view the blue stage from a different perspective.
It would therefore be an improvement over the state of the art to provide a new method for utilizing garbage mattes within a virtual set, where the method enables more practical creation, movement and repositioning of garbage mattes. It would also be an improvement to provide real-time manipulation of the garbage mattes which would enable rapid deployment of the system for use in modern virtual sets. Preferably, the garbage matte would obscure everything around the blue stage, both in front of and behind the television cameras. In this way, no matter where the television camera turns, if objects seen by the television camera include physical elements which are not intended to be seen (such as props), the garbage matte would hide those objects.
It would also be an improvement if the position and orientation of the garbage matte could be anchored to the blue stage's position and orientation. In this way, the garbage matte would not require readjustment when the blue stage's virtual location is changed within the virtual environment.